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f  the  Weapon  System  Acquisition  Reform  Act  of  2009  (WSARA,  enacted  May  22,  2009)  is 
to  have  any  lasting  effect,  the  behaviors  of  the  defense  acquisition  workforce  must  change. 
One  of  my  major  concerns  is  how  we  can  better  train  our  major  defense  acquisition  program 
(MDAP)  managers  and  support  staffs  in  the  practical  application  of  the  tenets  or  principles 
of  WSARA,  most  of  which  are  really  not  new;  they  just  mean  getting  back  to  the  basics  of 
acquisition!  This  article  addresses  three  key  challenges  of  WSARA  and  outlines  some  actions  we 
need  to  take  to  change  the  culture  of  our  acquisition  workers. 


Integrated  Cost  and  Schedule  Estimation 

First,  we  need  to  adopt  an  integrated  team  approach  to  cost  and  schedule  estimation.  For  too  long,  we  have  left 
cost  estimation  to  the  cost  estimators.  To  further  aggravate  the  situation,  we  outsourced  many  of  our  government 
cost  estimators  in  the  1990s  and  are  paying  the  price  today.  In  the  past,  we  have  expected  the  cost  estimators 
alone  to  do  the  business  of  cost  estimation,  yet  we  never  told  them  all  they  needed  to  know  in  order  to  prepare  a 
realistic  cost  estimate. 


Then  once  we  got  their  cost  estimate,  we  pressed  them  to  reduce  the  estimate  to  a  more  "affordable"  number. 
We  also  hoped  for  new  manufacturing  processes  and  economies  of  scale  that  might  keep  the  program  affordable 
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(for  example,  the  Joint  Strike  Fighter).  Sometimes,  we 
even  threw  out  the  cost  estimate  altogether  and  simply 
funded  to  available  budget.  Such  was  the  case  with  the 
Army's  Future  Combat  System  when  it  entered  develop¬ 
ment  at  Milestone  B  in  2003.  Not  a  good  way  to  start  a 
program!  According  to  Gene  Porter  in  a  December  2009 
Institute  for  Defense  Analyses  paper  entitled  "The  Major 
Causes  of  Cost  Growth  in  Defense  Acquisition,"  when 
that  occurs,  the  entire  decision-making  process  is  put 
at  risk,  including  both  the  original  analysis  of  alterna¬ 
tives  and  the  subsequent  stability  and  executability  of 
the  program. 

Cost  and  schedule  estimation  is  a  craft— a  craft  that 
requires  reasoned  inputs  from  systems  engineers,  lo¬ 
gisticians,  contracting  officers,  and  testers,  in  addition 
to  those  of  experienced  cost  estimators.  It  requires 
an  integrated  team  of  functional  experts  dedicated  to 
identifying  risk  and  assigning  cost  and  schedule  to  that 


risk.  Cost  and  schedule  estimates  cannot  be  done  in  a 
vacuum  by  a  single  estimator.  It's  a  team  sport  in  which 
multi-functional  inputs  are  essential  for  success. 

WSAR  A  created  the  position  of  director  of  cost  assess¬ 
ment  and  program  evaluation  in  2009.  In  addition  to 
reviewing  all  component  cost  estimates  and  conducting 
an  independent  cost  estimate  for  MDAPs,  the  director 
of  cost  assessment  and  program  evaluation  is  to  pro¬ 
vide  policies  and  procedures  for  all  DoD  cost  estimates. 
That's  a  tall  order,  and  one  that  can  be  achieved  only  if 
the  grassroots  acquisition  workers  make  integrated  cost 
and  schedule  estimation  part  of  their  day-to-day  routine. 
That's  because  no  policy  or  procedure  can  ever  be  writ¬ 
ten  that  will  turn  over  all  the  technical  and  programmatic 
"rocks"  under  which  cost  and  schedule  risks  lie  in  wait¬ 
ing.  Our  systems  are  just  too  complex.  And  even  if  it 
could  be  written,  no  policy  or  procedure  has  ever  seen 
100  percent  compliance. 


If  the  Weapon  System  Acquisition  Reform  Act  of  2009 


is  to  have  any  lasting  effect,  the  behaviors  of  the 
defense  acquisition  workforce  must  change. 


WSARA  also  requires  the  disclosure  of  the  confidence  lev¬ 
els  for  baseline  estimates  for  MDAPs.  Justification  must  be 
provided  if  the  cost  estimate  is  calculated  at  a  confidence 
level  less  than  80  percent.  Now,  the  law  doesn't  specify  how 
confidence  levels  are  calculated  or  explain  why  80  percent 
is  the  target,  as  opposed  to  90  percent  or  70  percent.  The 
intent  of  the  law  is  to  hedge  against  cost  overruns. 

Wouldn't  we  serve  the  same  purpose  if  we  used  integrated 
cost  and  schedule  estimation  to  uncover  technical  and  pro¬ 
grammatic  risks  and  covered  those  risks  at  the  beginning  of 
the  program  to  create  more  realistic  cost  and  schedule  esti¬ 
mates?  Wouldn't  risk-informed  cost  and  schedule  estimates 
be  more  easily  defended  through  the  budgeting  process  and 
to  Congress? 

The  solution,  from  where  I  sit,  is  to  teach  and  model  inte¬ 
grated  cost  and  schedule  estimating  to  the  grassroots  ac¬ 
quisition  workers.  We  have  totally  revamped  ourtrainingfor 
cost  estimators  and  put  them  into  their  own  career  track,  and 
we  must  not  stop  integrating  cost  and  schedule  estimation  in 
our  other  acquisition  courses.  In  addition,  risk  identification 
and  management  should  become  part  of  the  curriculum  so 
the  acquisition  worker  can  discover  technical  and  program¬ 
matic  risks  and  adjust  cost  and  schedule  estimates  to  miti¬ 
gate  them.  I'll  come  back  to  that  point  later. 

Competitive  Prototyping 

Let's  talk  about  the  "art"  of  competitive  prototyping.  I  call 
prototyping  an  art  because  it  is  part  of  a  program's  acqui¬ 
sition  strategy.  From  my  experience,  acquisition  strategy 
development  is  more  of  an  art  than  a  prescriptive  science. 
I  also  know  from  teaching  in  the  DAU  PMT  352  program 
management  office  course  that  competitive  prototyping  is 
not  well  understood.  In  that  course,  students  have  to  lay  out 
a  strategy  for  competitive  prototyping  prior  to  Milestone  B. 
My  experience  is  that  we  get  all  kinds  of  approaches,  many 
of  which  reveal  that  students  don't  understand  exactly  what 
a  developmental  prototype  is  and  how  competitive  prototyp¬ 
ing  might  be  used  in  the  technology  development  phase  prior 
to  Milestone  B. 

For  MDAPs,  WSARA  mandated  competitive  prototyping  of 
systems  or  critical  subsystems  before  Milestone  B  approval, 
unless  waived  by  the  milestone  decision  authority.  Moreover, 


even  if  competitive  prototyping  is  waived,  a  prototype  must 
be  produced  before  Milestone  B. 

Competitive  prototyping  isn't  new  to  the  Department  of  De¬ 
fense.  Even  before  WSARA,  John  Young,  then-under  sec¬ 
retary  of  defense  for  acquisition,  technology  and  logistics, 
made  it  policy  to  have  multiple  competitive  prototypes  in 
order  to  determine  the  maturity  of  the  technology  and  get 
a  better  cost  estimate  prior  to  Milestone  B.  Today,  the  Joint 
Air-Ground  Missile,  Joint  Lightweight  Tactical  Vehicle,  and 
Small  Diameter  Bomb  II  are  examples  of  programs  that  seem 
to  be  using  competitive  prototyping  with  some  success. 

Yet  there  are  also  the  failures— not  failures  in  the  sense  of 
program  failure,  but  failures  in  the  sense  that  competitive  pro¬ 
totyping  really  does  not  appear  to  have  been  cost-effective. 
Porter  argues  that  the  cost  of  developmental  prototypes  for 
the  Joint  Strike  Fighter  and  Littoral  Combat  Ship  only  added 
to  cost  growth  and  may  not  have  been  worth  the  effort. 

The  concept  of  competitive  prototyping  is,  indeed,  new  to 
many  of  today's  acquisition  workers  because  its  use  has 
been  cyclical.  The  idea  of  prototyping  aircraft  engine  and 
airframe  combinations  can  be  traced  back  some  20  years 
before  World  War  II  and  was  fairly  common  into  the  1950s. 
A  "fly-before-buy"  strategy  was  instituted  in  the  late  1960s 
by  David  Packard,  then-deputy  secretary  of  defense,  but  it 
fell  out  of  favor  by  the  late  1970s.  Once  again,  the  1986  Pack¬ 
ard  Commission  Report  emphasized  prototyping  before  full- 
scale  development  and  this  became  part  of  DoD  Instruction 
5000.2  in  1987.  However,  both  Porter  and  Jeffrey  A.  Drezner, 
author  of  a  1992  Rand  Corporation  research  report,  "The 
Nature  and  Role  of  Prototyping  in  Weapon  System  Develop¬ 
ment,"  point  out  that  the  nature  of  prototyping,  the  condi¬ 
tions  under  which  one  should  prototype,  and  the  benefits  of 
prototyping  remain  unidentified. 

Today's  acquisition  workers  need  to  rethink  and  relearn 
competitive  prototyping.  They  need  to  be  trained  on  how  to 
make  a  sound  business  case  for  competitive  prototyping— if 
one  actually  exists.  They  need  to  think  through  how  they  will 
manage  two  or  more  contractors  in  a  competition-sensitive 
environment.  And— back  to  the  cost  estimating  that  I  dis¬ 
cussed  earlier— they  need  to  know  how  to  convince  decision 
makers  in  the  programming  and  budgeting  processes  that 
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the  additional  cost  of  multiple  prototypes  is  worth  the  fund¬ 
ing.  In  addition,  they  need  to  understand  advanced  technology 
demonstrations  and  joint  capability  technology  demonstra¬ 
tions  that  have  long  been  in  the  domain  of  the  science  and 
technology  community  but  should  now  be  considered  as  viable 
prototyping  approaches  in  the  technology  development  phase. 

Even  more  difficult  for  acquisition  students  to  understand  is 
how  to  compete  at  the  critical  subsystem  level,  as  is  permitted 
by  WSARA.  Full-up  system  prototypes  are  clearly  impractical 
for  big  developments  such  as  aircraft  carriers  and  for  one-of- 
a-kind  satellites.  Acquisition  workers  not  only  need  to  under¬ 
stand  how  to  down-select  from  competing  subsystem-level 
prototypes,  but  they  also  need  to  understand  the  on-ramp 
processes  by  which  these  winning  components  are  integrated 
back  into  the  larger  system. 

I  advocate  that  case  studies,  written  around  programs  that 
have  used  competitive  prototypes  (whether  successful  or  un¬ 
successful)  be  injected  into  DAU  program  management  certi¬ 
fication  courses.  In  addition,  the  PMT  352  program  manage¬ 
ment  office  course  should  include  a  seminar  on  competitive 
prototyping  just  prior  to  the  exercise  in  which  students  develop 
an  acquisition  strategy  around  competitive  prototyping. 

Systems  Engineering  Decisions 

Now  let  me  turn  to  the  third  challenge  of  asking  the  right 
questions  and  making  the  tough  systems  engineering  de¬ 
cisions,  especially  during  preliminary  and  critical  design 
reviews.  As  a  quick  review,  the  preliminary  design  review 
defines  the  allocated  baseline  for  the  weapon  system,  and 
according  to  WSARA,  the  preliminary  design  review  (PDR) 
for  MDAPs  must  come  before  the  Milestone  B  decision  re¬ 
view.  Similarly,  the  critical  design  review  defines  the  product 
baseline  for  the  system  and  now  separates  the  two  major 
efforts  of  the  engineering  and  manufacturing  development 
phase:  (1)  integrated  system  design;  and  (2)  systems  ca¬ 
pability  and  manufacturing  process  demonstration.  Prior  to 
WSARA,  DoD  Instruction  5000.02  raised  the  importance  of 
these  reviews  by  requiring  post-PDR  and  post-critical  design 
review  assessments  by  the  milestone  decision  authority,  with 
decisions  from  those  assessments  documented  in  acquisition 
decision  memoranda. 

During  the  technology  development  phase,  WSARA  and 
DoD  Instruction  5000.02  require  that  MDAPs  conduct  a 
system-level  PDR:  "A  successful  PDR  will  inform  require¬ 
ments  trades;  improve  cost  estimation;  and  identifies  re¬ 
maining  design,  integration,  and  manufacturing  risks."  The 
cost-performance  trades  that  result  from  knowledge  gained 
during  competitive  prototyping  can  help  keep  the  program 
affordable  and  within  the  Milestone  A  component  cost  es¬ 
timate.  But  are  we  teaching  our  acquisition  workers  what 
questions  to  ask  at  the  PDR  about  design,  integration  and 
manufacturing  risks?  More  important,  are  we  really  train¬ 
ing  them  to  make  the  tough  decisions  regarding  cost  and 
performance  trades? 


According  to  DoD  Instruction  5000.02,  "The  project  shall 
exit  the  technology  development  phase  when  an  affordable 
program  or  increment  of  militarily  useful  capability  has  been 
identified;  the  technology  and  manufacturing  processes  for 
that  program  or  increment  have  been  assessed  and  demon¬ 
strated  in  a  relevant  environment;  manufacturing  risks  have 
been  identified;  a  system  or  increment  can  be  developed  for 
production  in  a  short  timeframe  (normally  less  than  five  years 
for  weapon  systems);  or,  when  the  MDA  decides  to  terminate 
the  effort."  That's  a  lot  to  ask!  Are  we  really  training  the  people 
who  staff  our  pre-MDAP  program  offices  to  make  those  as¬ 
sessments  and  recommendations? 

Too  often  in  the  past,  programs  have  entered  the  engineering 
and  manufacturing  development  phase  without  having  dem¬ 
onstrated  required  technologies  in  a  relevant  environment, 
which  is  defined  as  technology  readiness  level  (TRL)  6.  In 
last  year's  class  of  Nunn-McCurdy-breaching  programs,  root 
cause  analyses  identified  several  bad  actors.  Porter  reports 
that  when  the  Army's  Future  Combat  System  entered  system 
development  and  demonstration  in  2003,  24  out  of  31  of  the 
identified  critical  technologies  were  at  TRLs  below  6.  None 
of  the  20  critical  technologies  was  at  TRL  6  when  the  Joint 
Tactical  Radio  Systems-Ground  Mobile  Radio  entered  system 
development  and  demonstration  in  2002.  The  War-fighter 
Information  Network-Tactical  had  only  three  of  12  critical  tech¬ 
nologies  at  TRL  6  when  it  entered  systems  development  and 
demonstration  in  2003  (Porter,  p.  44). 

WSARA  now  requires  the  director  of  defense  research  and 
engineering  to  conduct  an  independent  assessment  of  the 
technological  maturity  and  integration  risk  of  the  critical 
technologies  of  MDAPs.  In  addition,  the  DDRE  is  to  develop 
knowledge-based  standards  to  measure  the  technological  ma¬ 
turity  and  integration  risk  of  critical  technologies  at  key  stages 
in  the  acquisition  process.  In  the  past,  the  program  manager 
was  responsible  for  technology  readiness  assessments  that 
were  based  upon  definitions  provided  in  the  Defense  Acquisi¬ 
tion  Guidebook. 

Inadequacies  in  initial  system  design,  systems  engineering, 
and  risk  assessment  at  the  front  end  of  the  program  continue 
to  translate  into  poor  cost  and  schedule  estimates  (Porter,  p. 
45-46).  We  continue  to  shortchange  early  system  engineering 
efforts  in  that  critical  timeframe  between  identification  of  the 
capability  gap  and  Milestone  B.  In  past  acquisition  workforce 
downsizing  efforts,  we  got  rid  of  key  government  engineers 
who  shepherded  the  transition  of  new  technologies  into  ac¬ 
quisition  programs,  so  now  we  have  lost  their  knowledge  of 
how  to  assess  technology  readiness  and  manage  technology 
transition  risks. 

We  also  do  a  poor  job  of  estimating  systemic  risks  inherent  in 
the  total  system  design.  As  we  link  systems  to  other  systems, 
government  program  management  office  personnel  need  to 
better  understand  the  integration  and  interoperability  chal¬ 
lenges.  Case  in  point:  We  don't  again  want  to  get  into  a  posi- 
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We  need  to  teach  our 
acquisition  workers  how 
early  systems  engineer¬ 
ing  design  reviews  can 
identify  risks.  We  need  to 
help  them  understand  the 
risks  associated  with  the 
integration  of  systems  of 
systems.  We  need  to  lead 
them  through  case  stud¬ 
ies  that  demonstrate  the 
value  of  early  systems 
engineering  and  teach 
them  some  of  the  basic 
questions  that  need  to 
be  asked.  We  also  need 
to  train  our  people  in  the 
early  decisions  that  must 
be  made  regarding  which 
technologies  are  ready 
for  the  first  increment  of 
development  and  which 
technologies  need  to  be 
deferred  to  later  incre¬ 
ments  of  capability. 


We  need  to  emphasize 
early  systems  engineering 
in  our  on-line  fundamental 
and  intermediate  systems 
acquisition  management 
courses.  We  also  need  to 
integrate  more  risk  man¬ 
agement  training  in  all  our 
acquisition  courses.  Cur¬ 
rently,  risk  management  is 
taught  only  as  a  targeted 
training  event  at  the  re¬ 
quest  of  a  program  office  or  acquisition  command.  Much 
can  be  done  to  make  our  risk  instruction  more  robust  and  to 
link  it  more  clearly  to  early  systems  engineering. 

Institutionalizing  WSARA 

What  will  it  take  to  really  institutionalize  WSARA?  I  feel 
strongly  that  changing  the  culture  of  the  acquisition  work¬ 
force  requires  that  we  change  the  way  we  teach  and  model 
the  acquisition  process.  I've  discussed  three  acquisition  chal¬ 
lenges  to  begin  with  as  we  seek  to  change  behaviors  and  get 
back  to  basics.  First,  we  need  an  integrated  team  approach  to 
estimating  cost  and  schedule.  Cost  and  schedule  estimation 
are  not  the  responsibility  of  the  cost  estimator  alone.  Second, 


Arnie,  whose  program  held  the  2010  record  for  cost  overruns,  is  inducted  into  the 

Acquisition  Hall  of  Shame. 


we  must  teach  the  art  of  competitive  prototyping;  we  must 
rethink  and  relearn  from  the  past  and  define  the  nature  of 
prototyping,  under  what  conditions  one  should  prototype, 
and  the  benefits  of  prototyping.  And  third,  we  must  help 
our  acquisition  workers  ask  the  right  questions  and  make 
the  tough  systems  engineering  decisions,  especially  during 
preliminary  and  critical  design  reviews.  Those  actions  will 
go  a  long  way  in  helping  us  understand  programmatic  and 
technical  risks  earlier. 

Fast  teaches  acquisition  and  program  management  courses  at  the  Naval 
Postgraduate  School.  Prior  to  that,  he  taught  in  the  program  management 
office  course  at  DAU.  The  author  welcomes  comments  and  questions  and 
can  be  contacted  at  wrfast(£)nps.edu. 


tion  where  we  have  to  hire 
a  lead  systems  integrator, 
as  was  the  case  with  the 
Army's  Future  Combat 
System. 
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